Camera-based Transparent Display

ABSTRACT

Presenting an image of a scene may include capturing an image of a scene by a camera of an electronic device, wherein the electronic device comprises the camera and a display, and wherein the camera and the display have a first spatial relationship, determining a second spatial relationship between a viewpoint and the display of the electronic device, warping the image to obtain an image of a first portion of the scene based on the first spatial relationship and the second spatial relationship, and presenting the warped image on the display, wherein, from the viewpoint, the image of the first portion of the scene is substantially contiguous with a second portion of the scene visible outside an edge of the electronic device.

BACKGROUND

This disclosure relates generally to the field of digital imageprocessing, and more specifically to the field of providing acamera-based transparent display.

Many multifunctional electronic devices, such as smart phones andtablets, include a camera and display device that allow a user toutilize the display to preview and view images captured by the camera.When a traditional multifunctional device is used in preview mode, theimage displayed on the screen represents an image from the camera'spoint of view. Therefore, when a user holds the device in front of ascene and views the display, the image on the screen does not line upwith the scene behind it. That is, the user's point of view differs fromthe point of view of the lens capturing the image, leading to adisconnect between the user's point of view and the real environment.That is, to the user, the image may appear in the wrong position and thewrong scale.

SUMMARY

In one embodiment, a method for presenting an image of a scene isdescribed. The method may include capturing an image of a scene by acamera of an electronic device, wherein the electronic device comprisesthe camera and a display, and wherein the camera and the display have afirst spatial relationship, determining a second spatial relationshipbetween a viewpoint and the display of the electronic device, warpingthe image to obtain an image of a first portion of the scene based onthe first spatial relationship and the second spatial relationship, andpresenting the warped image on the display, wherein, from the viewpoint,the image of the first portion of the scene is substantially contiguouswith a second portion of the scene visible outside an edge of theelectronic device.

In another embodiment, the method may be embodied in computer executableprogram code and stored in a non-transitory storage device. In yetanother embodiment, the method may be implemented in an electronicdevice having image capture capabilities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows, in block diagram form, a simplified electronic deviceaccording to one or more embodiments.

FIG. 2 shows, in flow chart form, an example method for displaying awarped image of a real environment, according to one or moreembodiments.

FIG. 3 shows, flow chart form, method for warping an image of a realenvironment in accordance with one or more embodiments.

FIG. 4 shows, in system diagram form, an example setup of using a deviceto view images of a real world environment, according to one or moreembodiments.

FIG. 5 shows an example system diagram illustrating a method ofdisplaying a warped image of a real environment, according to one ormore embodiments.

FIG. 6 shows, an example system diagram illustrating a warping method,according to one or more embodiments.

FIG. 7 shows, in block diagram form, a simplified multifunctional deviceaccording to one or more embodiments.

DETAILED DESCRIPTION

This disclosure is directed to systems, methods, and computer readablemedia for presenting a warped image of a scene. In general, techniquesare disclosed to provide a warped image of a scene such that the imageof the environment in the scene appears seamless with the realenvironment on a display from a viewpoint. According to one or moreembodiments, displaying the warped image may provide a more realisticview of the real environment on an electronic device.

According to one or more embodiments, a back-facing camera of anelectronic device may capture an image of a view of a real environment.A determination may be made regarding a spatial relationship between aviewpoint and the electronic device, and a spatial relationship betweena camera and a display of the electronic device. A warping function maybe applied to the image based on the spatial relationships. The warpedimage may then be presented on a front-facing display on the electronicdevice such that the image of the real environment appears substantiallycontiguous with background (i.e., the rest of the real environment).That is, the scene may appear contiguous in the real world and thedisplay, except for any borders or boundaries of the electronic devicearound the display. Given a viewpoint relative to a screen of anelectronic device, the image captured by the camera may be transformedrelative to the screen, or re-projected, so that at least one object inthe scene has the same position and sale as the real object behind thescene, from the viewpoint of the user. Thus, the electronic device mayappear transparent, or the image of the scene may appear substantiallyseamless with the scene behind it from the viewpoint of the user,according to one or more embodiments.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the disclosed concepts. As part of this description,some of this disclosure's drawings represent structures and devices inblock diagram form in order to avoid obscuring the novel aspects of thedisclosed embodiments. In this context, it should be understood thatreferences to numbered drawing elements without associated identifiers(e.g., 100) refer to all instances of the drawing element withidentifiers (e.g., 100 a and 100 b). Further, as part of thisdescription, some of this disclosure's drawings may be provided in theform of a flow diagram. The boxes in any particular flow diagram may bepresented in a particular order. However, it should be understood thatthe particular flow of any flow diagram is used only to exemplify oneembodiment. In other embodiments, any of the various components depictedin the flow diagram may be deleted, or the components may be performedin a different order, or even concurrently. In addition, otherembodiments may include additional steps not depicted as part of theflow diagram. The language used in this disclosure has been principallyselected for readability and instructional purposes, and may not havebeen selected to delineate or circumscribe the disclosed subject matter.Reference in this disclosure to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least oneembodiment, and multiple references to “one embodiment” or to “anembodiment” should not be understood as necessarily all referring to thesame embodiment or to different embodiments.

It should be appreciated that in the development of any actualimplementation (as in any development project), numerous decisions mustbe made to achieve the developers' specific goals (e.g., compliance withsystem and business-related constraints), and that these goals will varyfrom one implementation to another. It will also be appreciated thatsuch development efforts might be complex and time consuming, but wouldnevertheless be a routine undertaking for those of ordinary skill in theart of image capture having the benefit of this disclosure.

For purposes of this disclosure, the term “lens” refers to a lensassembly, which could include multiple lenses. In one or moreembodiments, the lens may be moved to various positions to captureimages with different points of focus. Further in one or moreembodiments, the lens may refer to any kind of lens, such as atelescopic lens or a wide angle lens. As such, the term lens can mean asingle optical element or multiple elements configured into a stack orother arrangement.

For purposes of this disclosure, the term “camera system” refers to oneor more lens assemblies along with the one or more sensor elements andother circuitry utilized to capture an image. For purposes of thisdisclosure, the “camera” may include more than one camera system, suchas a stereo camera system, multi-camera system, or other camera systemcapable of sensing the depth of the captured scene.

Referring to FIG. 1, a simplified block diagram of an electronic device100 is depicted, in accordance with one or more embodiments of thedisclosure. Electronic device 100 may be part of a multifunctionaldevice, such as a mobile phone, tablet computer, personal digitalassistant, portable music/video player, wearable device, or any otherelectronic device that includes a camera system. FIG. 1 shows, in blockdiagram form, an overall view of a system diagram capable of supportingimage warping, according to one or more embodiments. Specifically, FIG.1 depicts an electronic device 100 that is a computer system. Electronicdevice 100 may be connected to other network devices across a network,such as mobile devices, tablet devices, desktop devices, as well asnetwork storage devices such as servers and the like.

Electronic Device 100 may include processor 130, such as a centralprocessing unit (CPU). Processor 130 may be a system-on-chip such asthose found in mobile devices and include one or more dedicated graphicsprocessing units (GPUs). Further processor 130 may include multipleprocessors of the same or different type. Electronic Device 100 may alsoinclude a memory 140. Memory 140 may each include one or more differenttypes of memory, which may be used for performing device functions inconjunction with processor 130. For example, memory 140 may includecache, ROM, and/or RAM. Memory 140 may store various programming modulesduring execution, including warping module 155 and position detectionmodule 160.

Electronic device 100 may also include one or more cameras, such asfront-facing camera 110 and back-facing camera 120. Cameras 110 and 120may each include an image sensor, a lens stack, and other componentsthat may be used to capture images. In one or more embodiments, thecameras may be directed in different directions in the electronicdevice. For example, front-facing camera 110 may be positioned in or ona first surface of the electronic device 100, while the back-facingcamera 120 may be positioned in or on a second surface of the electronicdevice 100. In one or more embodiments, the first and second surfacesmay be opposite surfaces of the electronic device 100. As anotherexample, the back-facing camera may be configured to capture images ofan environment from the back of the electronic device 100, whereasfront-facing camera is configured to capture images of the user, orother objects in front of the device. Although two cameras are depicted,as will be described in further detail below, in one or moreembodiments, electronic device 100 may include a single camera, or mayinclude additional cameras, such as two back-facing cameras and onefront facing camera. In one or more embodiments, the electronic device100 may also include a display 150. Display 150 may be any kind ofdisplay device, such as an LCD (liquid crystal display), LED(light-emitting diode) display, OLED (organic light-emitting diode)display, or the like. In addition, display 150 could be a semi-opaquedisplay, such as a heads-up display or the like. Display 150 may belocated on the same surface of the electronic device 100 as thefront-facing camera 110.

Although electronic device 100 is depicted as comprising the numerouscomponents described above, in one or more embodiments, the variouscomponents may be distributed across multiple devices. Further,additional components may be used, some combination of the functionalityof any of the components may be combined.

In one or more embodiments, position detection module 160 is configuredto determine spatial relationships between a user and the device. Forexample, position detection module 160 may utilize sensor data, forexample from back-facing camera 120 or sensors 175 to detect a viewpointof a user. As an example, the position detection module 160 may track arelative position of a viewpoint compared to the device. In one or moreembodiments, the viewpoint may be a point in space from which the sceneis viewed via the display. For example, the viewpoint may be located ator near a user's eye or eyes, such as in the middle of the user's eyes,or at a location near the user's eyes. In one or more embodiments, theposition detection module 160 may track a relative pose of an eye to thedisplay 150 of the electronic device 100. That is, position detectionmodule 160 may implement eye tracking and/or gaze detection technology.In one or more embodiments, the viewpoint may be considered the positionin space at or near which a user's eye is located. The pose of theuser's eye and the relationship of the eye to the electronic device maybe determined using the front-facing camera 110 to capture an image ofthe user's eye. The camera capturing the image of the user's eye may bea traditional camera, including a lens stack and sensor, or may be alenseless camera. According to one or more embodiments, the relationshipbetween the pose of the eye and the display may be based on one or moreintermediary measurements. As an example, the position detection module160 may determine a relationship between the viewpoint and thefront-facing camera 110, or other component of the electronic device.From there, the position detection module 160 may determine the spatialrelationship between the viewpoint and the display 150 based on knownspatial relationships between the various components of electronicdevice 100. Further, in one or more embodiments, a sensor (e.g., depthsensor) may be used to determine a distance between the viewpoint andthe electronic device, as well as an angular position and orientationbetween the viewpoint and the electronic device.

In one or more embodiments, the position detection module 160 mayadditionally, or alternatively, utilize other sensor data from sensors175 capable of tracking movement and position of a viewpoint, such as auser's eye. The position detection module 160 may further track a targetat which an eye is gazing, according to one or more embodiments. A gazedetection method may be utilized to identify a region or object ofinterest in the real environment. As an example, the position detectionmodule 160 may determine a location on the display 150 at which a useris gazing, and identify an object or region in the real environmentassociated with that location.

According to one or more embodiments, warping module 155 may includeinstructions executable by a processor 130 to generate a warped image ofa real environment as captured by back-facing camera 120. In one or moreembodiments, the warping module may obtain an image of a realenvironment, for example, by back-facing camera 120. The warping module155 may generate a warped view of the real environment based on theviewpoint determined by the position detection module 160 such that theimage of the environment appears substantially contiguous or seamlesswith the real environment as viewed from the viewpoint. Said anotherway, in one or more embodiments, the image may be warped such that animage of the scene on the display of the electronic device appearssubstantially contiguous with the scene viewed around the electronicdevice from the viewpoint. For example, a first portion of the scene maybe visible from the viewpoint in the display, and may appearsubstantially contiguous with a second portion of the scene visiblearound the electronic device. The scene may appear substantiallycontiguous if the view of the scene is interrupted by a boundary of thedisplay and/or the electronic device from the viewpoint.

In one or more embodiments, the warping module 155 warps the image byback-projecting an image based on camera intrinsics in order to obtain a2D mapping for the 3D environment. The back-projection may be based on atarget depth. The target depth may be predetermined, or determined basedon a region of interest, or point of interest, in the image. Forexample, the depth may be selected based on a depth of an object ofinterest. In one or more embodiments, functionality of the electronicdevice 100 may be utilized to select a depth. As an example, in one ormore embodiments, the depth may be selected based on a detected subject,for example, using an autofocus operation of the electronic device 100.As another example, the warping module 155 may utilize sensor data todetect depth of field of the scene of the real environment. Sensor datamay be utilized in order to determine depth based on stereoscopicmethods, structured light, time of flight, and the like. Stereoscopicmethods may include utilizing a stereo camera system to determine depthin a field of view, such as through stereo estimation. A structuredlight operations may involve projecting a pattern onto a scene anddetermining a depth of the scene based on the appearance of theprojected pattern. Time of flight operations may include determiningdistance based on a determination of a time it takes for a wave totravel to an object in the scene. A depth map may either be determinedor obtained that may be used to determine a depth. For example, a depthmay be determined from a depth map for a particular point or region ofinterest. According to one or more embodiments, back-projecting theimage may include identifying four image corners, where the frustum fromthe camera intersects the determined depth. Thus, based on thedetermined depth, four points in space may be identified. The resultingplane may be determined to be the back-projected image.

In one or more embodiments, the warping module 155 further re-projectsthe back-projected image based on the determined relationship betweenthe viewpoint and the display. In one or more embodiments, there-projected image is displayed on the screen such that the image of thescene matches the real world “background” scene. Said another way, theimage is warped such that the plane defined by the four points isdisplayed on the display 150. Thus, the portion of the scene hidden bythe electronic device when viewed from the viewpoint, is visible in thedisplay. The remaining portion of the scene may be visible from theviewpoint in the real world surrounding the electronic device from theviewpoint. As another example, an object in the scene may be partiallyobfuscated by the electronic device when viewed from the viewpoint. Afirst portion of the object may be visible in the real environment fromthe viewpoint, and a second portion of the object may be visible in theimage of the environment shown in the display.

In one or more embodiments, the warping module may further warp theimage to better match the scene in the real environment. For example,the warping module may perform color correction on the image, such asperforming a white balancing function, such that the coloration of theimage better matches coloring in the real environment.

FIG. 2 shows, flow chart form, a method for providing a warped displayof a real environment. For purposes of explanation, the following stepswill be described in the context of FIG. 1. However, it should beunderstood that the various actions may be taken by alternatecomponents. In addition, the various actions may be performed in adifferent order. Further, some actions may be performed simultaneously,and some may not be required, or others may be added.

The flow chart begins at 205 where the warping module 155 obtains animage of a scene in a real environment. In one or more embodiments,warping module 155 may obtain the image from back-facing camera 120. Inone or more embodiments, the warping module 155 may obtain the imagefrom another source, such as an external camera, or a peripheral camera.An image may include a still image, a video stream, a series of frames,a live image, or the like.

The flow chart continues at 210, where the warping module 155 determinesa spatial relationship between a viewpoint and the device. In one ormore embodiments, the spatial relationship is obtained from the positiondetection module 160. According to one or more embodiments, the spatialrelationship may be based on a relationship between a location of aviewpoint, such as a location at or near one or both eyes of a user, andthe display 150 of electronic device 100. In one or more embodiments,the eye from which viewpoint is determined may be predetermined, forexample, based on a dominant eye for the user, or based on a locationbetween a user's two eyes, or the like.

According to one or more embodiments, the viewpoint may change overtime. For example, a user may move the electronic device relative totheir face. The viewpoint may be tracked, for example, with auser-facing sensor on the electronic device, such as front-facing camera110 or another sensor from the sensors 175 in the electronic device. Asan example, the point of view of the user may be determined based on adetection of the user's gaze, by performing head or eye tracking basedon sensor data from an RGBD sensor data. In one or more embodiments,sensor data may be used to track other facial landmarks, such as eyepositions, to determine a relative location of the user from the device,and the gaze direction of the user. As another example, an RGB onlysensor (that is, an RGB sensor that doesn't capture depth information),may be used to track head position, facial landmarks, and the like. Inone or more embodiments, depth information of the user may be determinedusing other means, such as a time-of-flight laser range sensor. Thedepth may also be inferred through characteristics in an image capturedby the front-facing camera of a user's face, such as the spacing betweenthe users' eyes, based on a user's inter-pupillary distance. The dataregarding the gaze direction of the user and the relative position ofthe user may be used to determine a viewpoint of the user, according toone or more embodiments.

The flow chart continues at 215, and the warping module 215 warps theimage based on the spatial relationship between the viewpoint and thedevice, along with the known intrinsics of the camera. In one or moreembodiments, the warping module 215 may back-project the image based onintrinsics of the camera capturing the image. Then, the warping module215 may re-project the back-projected image based on the determinedviewpoint.

The flowchart continues at 220 and the warping module 155 displays thewarped image. As described above, the warping module 155 may display asingle warped image based on a single determined viewpoint. Further, ifthe display 150 includes a stereo display, then warping module 155 maydisplay two warped images, each corresponding a viewpoint from each of auser's eyes, as an example.

At 225, a determination is made regarding whether movement is detected.In one or more embodiments, a determination is made regarding whethermovement of the viewpoint is detected in relation to the electronicdevice. If at 225 a determination is made that movement is not detected,then the warped area is continued to be displayed, as in at 220. In oneor more embodiments, continuing to display the warped image may includecontinuing to generate updated warped images based on updated image datareceived by the electronic device. Said another way, the image data maybe continued to be warped in a same manner if a movement between theviewpoint and the electronic device is not detected. Thus, for example,if the scene changes, or a relationship between the scene and theelectronic device changes, such as if the user pans the electronicdevice, the image may be continued to be warped in a same or similarmanner.

Returning to 225, if a determination is made that the movement isdetected, then the flow chart continues at 210, where the new spatialrelationship between the viewpoint and the device is determined. Thus,in one or more embodiments, the new spatial relationship between theviewpoint and the device may require the image to be warped in adifferent manner to appear substantially contiguous with the real worldenvironment.

Referring now to FIG. 3, a flow chart showing a method for warping animage of a scene in a real environment such that the image appearssubstantially contiguous or seamless with the real environment from aviewpoint, according to one or more embodiments. In one or moreembodiments, the various actions take place as part of warping the imagebased on the spatial relationship between the viewpoint and the device,and the known intrinsics of the device, as in 215 of FIG. 2. However,the various actions may take place in other locations within FIG. 2. Forpurposes of explanation, the following steps will be described in thecontext of the various components described in FIG. 1. However, itshould be understood that the various actions may be taken by alternatecomponents. In addition, the various actions may be performed in adifferent order. Further, some actions may be performed simultaneously,and some may not be required, or others may be added.

The flow chart begins at 305 and the warping module 155 selects a depthof a scene as the target depth. In one or more embodiments, in order totransform (or re-project) the camera image accurately may require 3Dinformation from every pixel in the image. However, the 2D camera imagemay no longer have any depth information. Therefore, a single depth maybe selected for every pixel in the image. In one or more embodiments,the depth may be selected based on a depth of the main subject in thescene, such as a person or object. In one or more embodiments, the depthmay be based on a location in the scene where a user may be looking.Selecting the depth of the main subject in the scene may ensure that themain subject the user is viewing has approximately the correct scale andposition in order to maintain the illusion of transparency of theelectronic device. In one or more embodiments, the depth of the mainsubject in the scene relative to the electronic device may be estimatedthrough any method that determines depth. For example, if theback-facing camera 120 actually includes two cameras, the depth of thescene may be estimated using stereo estimates from the pair ofscene-facing cameras. In another example, the depth of the scene may bedetermined using another sensor, such as a time-of-flight laser rangesensor, or other depth sensing technology.

The flow chart continues at 310, and the warping module back-projectsthe image based on the intrinsics of the camera and the selected depth.As depicted, in one or more embodiments, back-projecting the image mayinclude various actions. As shown, back-projecting the image mayinclude, at 315, identifying four image corners based on cameraintrinsics and the selected depth. At 320, the four image corners areintersected with a plane defined by the display device to obtain a 2Dimage transform. Then, at 325, the 2D image points are mapped to the 3Dscene based on the 2D image transformation.

The flow chart continues at 330, where the position detection module 160determines a viewpoint. For example, the viewpoint may be a point inspace from which a user is viewing the scene. In one or moreembodiments, the viewpoint may be a point in three space from which ascene is viewed, and may include directional information. Thedirectional information may identify a direction at which a user isgazing in the scene or on the display, for example. As depicted within330, in one or more embodiments, determining a viewpoint may includevarious actions. At 335, the position detection module 160 capturessensor data from a sensor facing the user. Then, at 340, the positiondetection module 160 detects the viewpoint based on the sensor data.

The flow chart continues at 345, where the warping module re-projectsthe image to the display device based on the viewpoint of the user andthe selected depth. The flow chart concludes at 350, where the warpingmodule adjusts color processing for the image. In one or moreembodiments, if the warping module 155 were able to determine depth ofthe scene (for example, if back-facing camera 120 were actually a stereocamera, or if sensors 175 included additional sensors that provideddepth information), then each pixel may be re-projected based on actualdepth information, and may appear more accurate than selecting a singledepth at 305.

According to one or more embodiments, if back-facing camera 120 includeda stereo camera, or multiple cameras, then the various cameras may havemultiple viewpoints. Thus, the various steps of FIG. 3 may be reproducedfor each camera. Further, according to one or more embodiments, each ofthe various steps of FIG. 3 may be completed for each eye. Thus, whendetermining the spatial relationship between the viewpoint and thedevice, the viewpoint of each eye may be determined at 210. Further, ifthe display 150 is a stereoscopic display, then at 220 the warped imagefor each eye may be displayed.

Referring now to FIG. 4, a system diagram is shown for an example setupfor utilizing a device for presenting a warped image of a realenvironment, according to one or more embodiments. FIG. 4 shows a user410 utilizing an electronic device 100 that includes a front-facingcamera 110, a back-facing camera 120. In one or more embodiments, theuser may view a real environment 405 through the front-facing camera 110of the electronic device 100 from a viewpoint. In one or moreembodiments, an image of the real environment 405 includes a field ofview that is similar to a user's field of view 425 if the user were tolook directly at the real environment 405.

FIG. 5 shows an example system diagram of an augmented reality device towarp images, according to one or more environments. FIG. 5 shows threepoints in time during which various portions of embodiments arepracticed. At 505, a first version of the electronic device 100 isshown. The electronic device 100 includes a display 150, which may showa preview image 530A of two blocks 535 and 540, in a real environment.At 505, electronic device 100 captures the image 530A of the realenvironment, including the two blocks 535 and 540. The image 530A may becaptured by back-facing camera 120.

At 510, in one or more embodiment, the position detection module maydetermine a viewpoint of user 410. In one or more embodiments, theviewpoint may be detected using a front-facing camera 110, or some othersensor on the electronic device 100. In addition, in one or moreembodiments, the warping module 155 also back-projects the image 530A tothe selected depth 545. In one or more embodiments, the selected depthmay be determined using a set depth, or based on a detected object inthe scene. For example, the depth may be determined using an auto-focusor similar functionality to determine a depth of a particular point ofinterest in the scene. The image may be back-projected by intersectingthe four points that define the camera view frustum, and a plane definedby the selected depth. The warping module then re-projects the imagepack to the display device based on the viewpoint of the user 410 andthe selected depth 545.

At 515, the example flow diagram depicts an updated image 530B on theelectronic device 100 from the viewpoint of the user 410. As shown,blocks 535 and 540 both appear as they would if the display device wasnot in the way, as is apparent by the fact that block 540 is onlypartially shown in the image 530B, but the rest of the block is stillvisible, and appears seamless, in the real environment in the viewadjacent to the device. According to one or more embodiments, the sceneappears seamless in the image and the real environment if the objects inthe image appear at substantially the same scale and location as if theuser were looking at the scene from the same viewpoint but not throughthe display. In one or more embodiments, the image appears seamless ifthe environment in the image appears substantially continuous with thereal environment from the viewpoint. For example, the image may besubstantially continuous with the real environment if the user views thereal environment at the same scale and distance through the electronicdevice as they would if they electronic device were not present. Thus,any frame around the display or the electronic device may interfere withthe view of the real environment, but the image may still be consideredseamless or substantially continuous with the background realenvironment.

Referring now to FIG. 6, an example system diagram illustrating awarping method is presented, according to one or more embodiments. At600 a user at a viewpoint 410 viewing a scene that includes blocks 535and 540 through an electronic device 100. At 600, the four corners ofthe frustum from the camera to the determined depth of the scene areidentified. As described above, the depth may be determined using anumber of methods, such as an object detection technique, auto focus,user input, or the like. The four corners of the frustum are shown as602A, 602B, 602C, and 602D. The plane defined by the four corners 602may then be back-projected onto the display of electronic device 100. Inone or more embodiments, back-projecting the image captured within theplane may be back-projected based on a spatial relationship between thecamera and the display of the electronic device 100. In one or moreembodiments, the spatial relationship between the camera and the displaymay be known or encoded, for example by the electronic device 100, ormay be otherwise determined.

Turning to 610, the image plane defined by the four corners 602 isreprojected onto the display of the electronic device 100. In one ormore embodiments, the reprojection is based on the viewpoint, forexample the location and gaze direction of an eye or eyes of a user 410.Thus, the reprojection includes four corners 604A, 604B, 604C, and 604D,based on the frustum from the viewpoint to the plane. In one or moreembodiments, reprojecting the image generates an image for display inthe electronic device 100. In one or more embodiments, by reprojectingthe image based on the viewpoint, the scene including blocks 535 and 540may result in the blocks 535 and 540 appearing at the same scale andlocation through the electronic device 100 as they appear in the realworld environment. For example, if block 540 is partially occluded bythe electronic device from the gaze direction and location of theviewpoint, a portion of the block 540 that is occluded will appear inthe display of the electronic device 100 as it would if the electronicdevice 100 were not occluding the portion of the image. Further, in oneor more embodiments, the occluded portion and non-occluded portion ofblock 540 may appear substantially continuous from the viewpoint. As anexample, block 540 may appear continuous except for the border of theelectronic device 100 around the display.

Referring now to FIG. 7, a simplified functional block diagram ofillustrative multifunction device 700 is shown according to oneembodiment. Multifunction electronic device 700 may include processor705, display 710, user interface 715, graphics hardware 720, devicesensors 725 (e.g., proximity sensor/ambient light sensor, accelerometerand/or gyroscope), microphone 730, audio codec(s) 735, speaker(s) 740,communications circuitry 745, digital image capture circuitry 750 (e.g.,including camera system) video codec(s) 755 (e.g., in support of digitalimage capture unit), memory 760, storage device 765, and communicationsbus 770. Multifunction electronic device 700 may be, for example, adigital camera or a personal electronic device such as a personaldigital assistant (PDA), personal music player, mobile telephone, or atablet computer.

Processor 705 may execute instructions necessary to carry out or controlthe operation of many functions performed by device 700 (e.g., such asthe generation and/or processing of images as disclosed herein).Processor 705 may, for instance, drive display 710 and receive userinput from user interface 715. User interface 715 may allow a user tointeract with device 700. For example, user interface 715 can take avariety of forms, such as a button, keypad, dial, a click wheel,keyboard, display screen and/or a touch screen. Processor 705 may also,for example, be a system-on-chip such as those found in mobile devicesand include a dedicated graphics processing unit (GPU). Processor 705may be based on reduced instruction-set computer (RISC) or complexinstruction-set computer (CISC) architectures or any other suitablearchitecture and may include one or more processing cores. Graphicshardware 720 may be special purpose computational hardware forprocessing graphics and/or assisting processor 705 to process graphicsinformation. In one embodiment, graphics hardware 720 may include aprogrammable GPU.

Image capture circuitry 750 may include two (or more) lens assemblies780A and 780B, where each lens assembly may have a separate focallength. For example, lens assembly 780A may have a short focal lengthrelative to the focal length of lens assembly 780B. Each lens assemblymay have a separate associated sensor element 790. Alternatively, two ormore lens assemblies may share a common sensor element. Image capturecircuitry 750 may capture still and/or video images. Output from imagecapture circuitry 750 may be processed, at least in part, by videocodec(s) 755 and/or processor 705 and/or graphics hardware 720, and/or adedicated image processing unit or pipeline incorporated withincircuitry 765. Images so captured may be stored in memory 760 and/orstorage 765.

Sensor and camera circuitry 750 may capture still and video images thatmay be processed in accordance with this disclosure, at least in part,by video codec(s) 755 and/or processor 705 and/or graphics hardware 720,and/or a dedicated image processing unit incorporated within circuitry750. Images so captured may be stored in memory 760 and/or storage 765.Memory 760 may include one or more different types of media used byprocessor 705 and graphics hardware 720 to perform device functions. Forexample, memory 760 may include memory cache, read-only memory (ROM),and/or random access memory (RAM). Storage 765 may store media (e.g.,audio, image and video files), computer program instructions orsoftware, preference information, device profile information, and anyother suitable data. Storage 765 may include one more non-transitorycomputer-readable storage mediums including, for example, magnetic disks(fixed, floppy, and removable) and tape, optical media such as CD-ROMsand digital video disks (DVDs), and semiconductor memory devices such asElectrically Programmable Read-Only Memory (EPROM), and ElectricallyErasable Programmable Read-Only Memory (EEPROM). Memory 760 and storage765 may be used to tangibly retain computer program instructions or codeorganized into one or more modules and written in any desired computerprogramming language. When executed by, for example, processor 705 suchcomputer program code may implement one or more of the methods describedherein.

According to one or more embodiments, camera-based transparent displaymay have numerous uses. For example, the techniques described above maybe used for image stabilization. That is, a warping function utilized towarp the image may also be utilized to stabilize an image. In anotherembodiment, the camera-based transparent display may be used for machinelearning. For example, a camera may capture images of a scene andclassify the scene, and/or objects within the scene. In one or moreembodiments, by tracking an object at which a user's eye is gazing,objects may be identified which should be classified.

In one or more embodiments, the camera-based transparent display may beused for presenting virtual information on the display, for example foraugmented reality purposes. As an example, digital informationassociated with a particular location may be displayed in a moreaccurate location as perceived from a particular viewpoint through thedisplay. As an example, an image of the scene may be enhanced byadditional text or image data at a particular point within the scene inthe warped image. In one or more embodiment, the digital information maybe associated with a particular object in the scene. When the objectfalls within the warped image (or, when the object is visible in thedisplay), then the digital information associated with the object may bedisplayed along with the object in the warped image. In one orembodiment, as the warped image changes, whether a change in the sceneoccurs, or a change in the warping occurs due to a change in location ofthe viewpoint or the device relative to the scene, the digitalinformation may also change.

The scope of the disclosed subject matter should be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled. In the appended claims,the terms “including” and “in which” are used as the plain-Englishequivalents of the respective terms “comprising” and “wherein.”

1. A method for presenting an image of a scene, comprising: capturing animage of a scene by a camera of an electronic device, wherein theelectronic device comprises the camera and a stereoscopic display, andwherein the camera and the stereoscopic display have a first spatialrelationship; determining a second spatial relationship between a firsteye and the stereoscopic display of the electronic device; warping theimage to obtain a first warped image based on the first spatialrelationship and the second spatial relationship; determining a thirdspatial relationship between a second eye and the stereoscopic display;warping the image to obtain a second warped image based on the firstspatial relationship and the third spatial relationship; and presentingthe first warped image and the second warped image on the stereoscopicdisplay.
 2. The method of claim 1, wherein the first eye and the secondeye correspond to a viewpoint, and wherein, from the viewpoint, theimage of the scene is substantially contiguous with the scene visibleoutside an edge of the stereoscopic display.
 3. The method of claimwherein warping the image further comprises: back-projecting the imagebased on intrinsics of the camera and a depth in the scene.
 4. Themethod of claim 3, further comprising: selecting the depth using atleast one selected from a group consisting of an autofocus operation, astereo estimation, a time of flight operation, and a structured lightoperation; and back-projecting the image based on the selected depth. 5.The method of claim 3, wherein back-projecting the image comprises:identifying four image corners, based on the intrinsics of the cameraand the depth; and intersecting the four image corners with a planedefined by the display to obtain a 2D image transform.
 6. The method ofclaim 1, further comprising: determining a gaze direction of the firsteye, wherein the second spatial relationship is based on the gazedirection of the first eye.
 7. The method of claim 1, wherein warpingthe image further comprises performing a white balancing operation onthe image.
 8. The method of claim 1, further comprising: identify anobject in the scene; and wherein warping the image to obtain the firstwarped image comprises warping the image such that a first portion ofthe object is visible in the scene from a viewpoint of the first eye,and an image of a second portion of the object is visible on the displayfrom the viewpoint of the first eye, and wherein the first portion andthe second portion are substantially aligned.
 9. A non-transitorycomputer readable medium comprising computer readable code forpresenting an image of a scene, executable by one or more processors to:capture an image of a scene by a camera of an electronic device, whereinthe electronic device comprises the camera and a stereoscopic display,and wherein the camera and the stereoscopic display have a first spatialrelationship; determine a second spatial relationship between a firsteye and the stereoscopic display of the electronic device; warp theimage to obtain a first warped image based on the first spatialrelationship and the second spatial relationship; determine a thirdspatial relationship between a second eye and the stereoscopic display;warp the image to obtain a second warped image based on the firstspatial relationship and the third spatial relationship; and present thefirst warped image and the second warped image on the stereoscopicdisplay.
 10. The non-transitory computer readable medium of claim 9,wherein the first eye and the second eye correspond to a viewpoint, andwherein, from the viewpoint, the image of the first portion of the sceneis substantially contiguous with a second portion of the scene visibleoutside an edge of the stereoscopic display.
 11. The non-transitorycomputer readable medium of claim 9, wherein the computer readable codeto warp the image further comprises computer readable code to:back-project the image based on intrinsics of the camera and a depth inthe scene.
 12. The non-transitory computer readable medium of claim 11,further comprising computer readable code to: select the depth using atleast one selected from a group consisting of an autofocus operation, astereo estimation, a time of flight operation, and a structured lightoperation; and back-project the image based on the selected depth. 13.The non-transitory computer readable medium of claim 11, wherein thecomputer readable code to back-project the image comprises computerreadable code to: identify four image corners based on the intrinsics ofthe camera and the depth; and intersect the four image corners with aplane defined by the display to obtain a 2D image transform.
 14. Thenon-transitory computer readable medium of claim 9, further comprisingcomputer readable code to: determine a gaze direction of the first eye,wherein the second spatial relationship is based on the gaze directionof the first eye.
 15. The non-transitory computer readable medium ofclaim 9, wherein warping the image further comprises performing a whitebalancing operation on the image.
 16. The non-transitory computerreadable medium of claim 9, further comprising computer readable codeto: identify an object in the scene; wherein the computer readable codeto warp the image to obtain the first warped image comprises computerreadable code to warp the image such that a first portion of the objectis visible in the scene from a viewpoint of the first eye, and an imageof a second portion of the object is visible on the display from theviewpoint of the first eye, and wherein the first portion and the secondportion are substantially aligned.
 17. A system comprising: one or moreprocessors; and one or more computer readable media comprising computerreadable code for presenting an image of a scene, executable by the oneor more processors to: capture an image of a scene by a camera of anelectronic device, wherein the electronic device comprises the cameraand a stereoscopic display, and wherein the camera and the stereoscopicdisplay have a first spatial relationship; determine a second spatialrelationship between a first eye and the stereoscopic display of theelectronic device; warp the image to obtain a first warped image basedon the first spatial relationship and the second spatial relationship;determine a third spatial relationship between a second eye and thestereoscopic display; warp the image to obtain a second warped imagebased on the first spatial relationship and the third spatialrelationship; and present the first warped image and the second warpedimage on the stereoscopic display.
 18. The system of claim 17, whereinthe first eye and the second eye correspond to a viewpoint, and wherein,from the viewpoint, the image of the first portion of the scene issubstantially contiguous with a second portion of the scene visibleoutside an edge of the stereoscopic display.
 19. The system of claim 17,wherein the computer readable code to warp the image further comprisescomputer readable code to: back-project the image based on intrinsics ofthe camera and a depth in the scene.
 20. The system of claim 19, furthercomprising computer readable code to: select the depth using at leastone selected from a group consisting of an autofocus operation, a stereoestimation, a time of flight operation, and a structured lightoperation; and back-project the image based on the selected depth.